Skip to content

JS parser (7.3) and the x86-64 validator spec (J.6) - #195

Open
atiernan wants to merge 61 commits into
claude/lir-specfrom
claude/parser-codecheck
Open

atiernan wants to merge 61 commits into
claude/lir-specfrom
claude/parser-codecheck

Conversation

@atiernan

@atiernan atiernan commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Two pieces of Phase 7 work in one branch, on top of #191:

7.3 The JS parser (crates/clazel-js-syntax)

A recursive-descent parser over the lexer, with automatic semicolon insertion, early errors and the depth limit.

  • Proven (Verus, 85 verified, 0 errors): check::finish returns an AST only if it is well formed, every span is inside the source and inside its parent, and its height is at most MAX_DEPTH.
  • Not proven, tested only: that the AST is the parse the spec defines (is_valid_parse), and that Err means no parse exists. The crate stays at tier 0 and does not yet implement api::JsParser.
  • Grammar rules proven (derive.rs): a node of the shape the parser builds derives by spec::d for leaves, the binary-operator levels, unary, conditional and assignment expressions, parentheses and . member access. check_leaf checks parsed leaves against the tokens. None of this is yet applied to the whole arena, and token-level claims assume the lexer meets its spec.
  • §16.2.1.1 export early errors: specified (spec commit first, with examples and two certified mutations), then implemented.
  • Test262: every positive test parses. 167 negative tests are accepted and listed in tests/expected_failures.txt, which can only shrink: all are regex contents (no clazel-js-regexp yet).
  • api::JsParser is not implemented: its ensures needs the full parse-correctness proof, and a wrapper would need a banned escape hatch.
  • Plus about 450 hand-written cases, golden AST shapes, and deep- and wide-nesting tests that never overflow the stack.

J.6 The x86-64 validator spec (crates/clazel-codecheck, new)

The spec and contract only, with no validator implementation yet, so tier 0 (target 2).

  • What a blob must be: canonical clazel_x64 encodings of the permitted instructions, one after another.
  • The policy it must meet:
    • each region access is preceded by a fixed bounds-check sequence with a Spectre mask;
    • context-block stores go only to the exit area;
    • r15 and rsp are never written;
    • every jump target is the start of an instruction or sequence;
    • the blob ends in ret or jmp.
  • Proven: 32 verified, 0 errors; 9 of 9 mutations killed with certificates.
  • ABI choices the validator fixes and the back end (J.9) must follow: the context register, the region table layout, and the bounds-check and exit sequences. They are listed at the top of spec.rs for review.
  • Open question for jit.md: a backward jmp can loop forever. The plan does not say how JIT-compiled loops give control back.

Checks

  • cargo xtask check passes. Tests and clippy pass for both crates.
  • cargo xtask verify --changed was not run: this changes Cargo.lock, so it would run the whole suite. CI's full verification is the check.

🤖 Generated with Claude Code

atiernan and others added 15 commits November 18, 2026 12:43
Expressions with the cover grammars read as the refined productions, statements, functions, classes, patterns and modules, with ASI and a depth counter. Early errors and the proven arena check follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e depth limit (7.3)

finish checks the arena the recursive-descent parser built in three linear passes (shape and spans, parents, heights) and is proven, in Verus, to return an Ast only if ast_wf, spans_wf and height <= MAX_DEPTH hold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(7.3)

A walk over the arena with the environment of each node mirrors spec::early_local and child_env. Tests: unit tests, a corpus of accepted and rejected programs, and the vendored Test262 (every positive test parses; the negative ones not rejected are listed in tests/expected_failures.txt and can only shrink).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Functions and classes count towards the nesting limit, a node's binding strength is remembered instead of walked, and tests cover deeply nested and very long sources and the exact shape of the AST (children layout and spans) for small programs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…differs (7.3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er of children (7.3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New crate, tier 0 (target 2). spec.rs defines which blobs are accepted:
the canonical encodings of clazel-x64 instructions, scanned once into
units (exit sequence, bounds-check sequence, single instruction) with
register facts, such that every region access goes through a loaded base
and a checked, masked index, the context register and rsp are never
written, every jump target and entry is a unit start, ret occurs only in
the exit sequence and the blob ends in ret or jmp. The ABI it fixes
(r15 context, region table, exit area) is listed in the header, since
jit.md leaves it open. contract.rs and api.rs give the validator's
requires/ensures: sound, and complete given clazel-x64's
canonical_prefix_free. No implementation, no safety theorem yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fifteen examples evaluate the spec (accepted: a checked load, a loop, an
exit, a second entry, an exit-operand store; refused: unchecked or
too-short or wrong-region index, writes to rsp and r15, jumps into a
check or over one, a store to the region table, falling off the end, a
bare ret, an entry or jump out of a unit) and one ties an exit sequence
to its bytes. Four general lemmas (region accesses are checked, the
table is never written, no reserved write or bare ret, a blob ends in
ret or jmp) are proven. Nine mutations: seven killed with certificates,
two (jump targets, facts forgotten at targets) only incidentally, since
their examples use compute_only and cannot be certificates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two more lemmas about the spec: every target and entry is a unit start,
and what is known about registers does not flow into a jump target. The
two mutations that had only been killed incidentally by compute_only
examples now carry kill certificates, so all nine mutations are killed
by a theorem.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The static block now rejects `await` and `for await` (§15.7.1
ContainsAwait), as spec::early_local does, so static-init-invalid-await
leaves the expected failures. The absent import parts stay where they
were; span_ok now allows them there. Adds accept and reject cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@atiernan
atiernan added this pull request to stack #188 November 18, 2026 16:08
atiernan and others added 6 commits November 18, 2026 16:09
…he parser builds (7.3)

lemma_leaf_literal and lemma_binary_derives: a node of the shape the parser builds, over the tokens it consumed, is derived by spec::d.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tokens (7.3)

Unary, conditional, assignment, parenthesized and member-dot rules derive what the parser builds; check_leaf checks a literal or name leaf against the executable tokens and proves spec::d from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Duplicate ExportedNames, ExportedBindings that are not declared, and ill-formed StringLiteral module export names, with a proven example each and two certified mutations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…7.3)

The parser follows the new rules of spec::early_local; the twelve Test262 module-code tests that listed them now pass, so only the regular expression tests remain expected failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The duplicate-name and lone-surrogate rules are written with quantifier-free helpers (names_repeat_from, surrogate_ok_at) so that their certified mutations can be evaluated by compute_only; both mutants are killed by a theorem.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
atiernan and others added 7 commits November 18, 2026 17:14
…heck needs (7.3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gainst the tokens (7.3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…kinds (7.3)

check_derivation returns Verified only if derives_root holds, proven in Verus: literals, parenthesized expressions, the binary operators + - * / % < >, expression statements and empty statements under a Script or Module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parser records the tokens it consumes and where it inserts a ;, and parse_checked runs check_derivation_canon on the arena. rustfmt and clippy are clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
atiernan and others added 7 commits November 18, 2026 22:35
…and debugger against the tokens (7.3)

Proven in arena_d: each statement's node is derived by spec::d in the root's
context, with the ; of do-while taking Ins::DoWhile. Verus: 221 verified. Verified
share of the positive Test262 runs 636 to 665 of 1568.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each node is checked in the context its parent's production gives it
(Done.cxs, Cx5): inv records node_done in cx_of(cxs[j]), and a parent's check
requires cx_ok (its own context and its children's are the one it passes down),
which the driver compares with kids_cx. The assignment (assign_ctx) is untrusted;
a wrong guess is Unsupported. Identifiers now read [Yield]/[Await] from the
node's context rather than the root's. No new kinds are covered yet. Verus: 230
verified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
encode_exec (inside verus!) writes the canonical encoding of an
instruction of the subset and is proven equal to the spec's `encode` for
every form, composed from verified pieces: push_u32/push_i32/push_u64
(little-endian immediates, equal to `le_bytes`), push_rex, enc_rr_exec
and enc_mem_exec (equal to `enc_rr`, `enc_rm_mem`). No external_body or
assume_specification; arithmetic uses saturating ops to satisfy the
workspace lints, with the proof that none saturates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extract_exec is proven equal to the spec's `extract` (helpers for the
byte reader, the little-endian words, the REX head, the ModR/M and SIB
fields, each equal to its spec function), and decode_exec re-encodes with
encode_exec and accepts only if the bytes begin with exactly that. It is
proven to meet contract::decode_ensures: a Some is the canonical
encoding of a valid instruction, and a None means no instruction's
encoding begins the bytes (by the completeness of `extract` and the
verified encoder). Verus on clazel-x64: 105 verified, 0 errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decode_all turns a blob into the instruction list the spec quantifies
over, with each instruction's start offset, and is proven to establish
`spec::decodes_to` and `start_of` for what it returns, using clazel-x64's
verified decode_exec (progress: every encoding has at least one byte).
Verus on clazel-codecheck: 36 verified, 0 errors. This is the decoding
half of the validator's soundness; the scan against the recursive spec
`scan`, and the None direction, are not done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The forall over 0 <= k + 1 < len took in k == -1, whose ks[k] is unconstrained, so a
FormalParameters or binding pattern could not be derived at all. It is now a
recursion over the positions (computable, so a mutation can be certified).
example_rest_ends_its_list proves a lone rest is a list and a rest before another
item is not; the mutation rest-anywhere-in-list is killed by it. Verus: 232
verified; mutate: 6 killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
atiernan and others added 4 commits November 18, 2026 23:16
unit_exec.rs: plain_ok_e, exit_at_e and check_at_e, each proven equal to
its spec function (`plain_ok`, `exit_unit_at`, `check_unit_at` with
`check_params`), over an exec array of 16 facts related to the spec's
list of bindings by `rel`, with `access_ok_e`, `written_gpr_e`,
`mem_access_e` and `base_load_region_e` equal to theirs. No escape
hatches; saturating/wrapping ops for the workspace lints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scan_exec walks the instructions with the exec fact array and, using the
verified unit checks, is proven to compute exactly the spec's recursive
`scan`: it returns None iff `scan(ins, tg, 0, no_facts())` is None, and
otherwise the per-offset flag "a unit starts here", equal to membership in
the spec's list of unit starts. lemma_scan_step and lemma_scan_refuses
carry the loop invariant scan(0) == acc + scan(j, facts). Also
lemma_start_of_lt (offsets increase).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… tokens (7.3)

Function declarations and expressions (generator, async, async generator),
FormalParameters (names, defaults, ...rest), FunctionBody, arrow and async arrow
functions, return, yield and await, each part in the context its production gives
it (cx_fn, the concise body's, the async arrow's). Verus: 261 verified; mutate: 6
killed. Verified share of the positive Test262 runs 665 to 765 of 1568; harness
files 3 to 7 of 41.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
atiernan and others added 4 commits November 18, 2026 23:33
validate.rs: `validate`, in verus!, is proven to return true only if the
blob is `spec::accepted` for the entries. It decodes with decode_all,
marks entries and jump targets (checked arithmetic, range-checked),
runs the proven scan_exec, and checks every target is a unit start;
lemmas: seq_has_at, subset_intro, enc_all_len, start_of_ge,
jump_targets_step. It replaces the plain-Rust exec.rs; tests/validator.rs
is kept and now runs against the verified validator and clazel-x64's
verified encoder and decoder (17 pass). Verus on clazel-codecheck: 64
verified, 0 errors. Completeness is not proven yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e inside

acc_lemmas.rs: seq_has of a concatenation, subset elimination, every unit
start the scan finds is an instruction offset, and an accepted blob's
entries and jump targets lie in [0, blob length). uniq.rs: start offsets
of a prefix, instruction k is encoded at its start offset, the next
instruction after a decoded prefix, and the end of the blob. decode_all
now ensures that if the blob decomposes as w at all, it returns Some with
exactly w (by clazel-x64's proven prefix-freeness).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ontract

`validate` now ensures contract::validate_ensures in full: it returns
true if and only if the blob is accepted for the entries. Every early
false is proven to mean no accepted decomposition: a decode failure (the
decomposition is unique), an empty list or one not ending in ret or jmp,
an entry or jump target outside the blob (acc_lemmas), a refused unit
(scan_exec), or a target that is not a unit start. Verus on
clazel-codecheck: 74 verified, 0 errors; mutate: 9 of 9 killed by a
theorem. The 17 tests pass against the verified validator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… list

CI's rustfmt step (cargo fmt --all -- --check) failed on every OS for
PR 195. Formatting only; no code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
random_bytes_never_panic took 401 s under Miri in CI, over xtask's 300 s
per-test limit, failing "Verify (Miri, clazel-codecheck)". Under Miri it now
runs the first 2 000 rounds of the same sequence (about 40 s), as
docs/VERIFICATION.md asks; the normal test job still runs all 20 000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Verify (Miri, clazel-js-syntax)" was killed twice ("the runner has received
a shutdown signal") in tests/deep.rs: deeply_nested_is_rejected_not_a_crash
(25 sources of 100 000 levels) and long_flat_programs_parse (600 000 to
900 000 characters) cannot finish under Miri, where one 2 000-token chain
takes 207 s. what_the_check_costs times two 1.7 million character parses.
These three are ignored under Miri with their reason, as docs/VERIFICATION.md
allows when one case takes minutes; a_chain_past_the_limit_is_too_deep and
parser.rs's depth_limit still cover TooDeep there.

Behind them, tests/order.rs read the test262 harness directory, which Miri's
isolation forbids. That part is now ignored under Miri ("reads files"), and
its inline sources are a separate test that still runs there.

Locally, the whole crate under Miri (nightly-2026-08-21) passes; the
slowest test is scripts_that_parse at 226 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@atiernan
atiernan temporarily deployed to full-verification November 20, 2026 17:18 — with GitHub Actions Inactive
@atiernan

Copy link
Copy Markdown
Member Author

Closing and reopening to start CI: no run was created for 3af71a5.

@atiernan atiernan closed this Nov 20, 2026
@atiernan atiernan reopened this Nov 20, 2026

This branch was previously deployed

1 inactive (outdated) deployment
full-verification — 6f4d5210 Deployed Nov 20, 2026 by atiernan via Approve the proofs #611
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant